SALR: A Sparsity-Aware Low-Rank Fine-Tuning Method That Reduces LLM Model Size by 2x While Maintaining Performance

Adapting large pre-trained language models to downstream tasks often entails fine-tuning millions of parameters or deploying costly dense weight updates, which hinders their use in resource-constrained environments. Low-rank Adaptation (LoRA) reduces trai